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value, each one pixel value is output, as shown in Fig. 8 IB. The pixel value output by 
the pixel E corresponds to the integrated value of the pixel value f (x) in the range of 
the light reception area. 

The correlation calculating unit 703 generates the correlation value between the 
pixel value of the pixel D and that of the pixel E, and the correlation value between the 
pixel value of the pixel E and that of the pixel F, to route the so-generated correlation 
value to the pixel value; generator 702. The correlation value calculated by the 
correlation calculating urit 703 is calculated based on the difference between the pixel 
value of the pixel D and that of the pixel E, or on the difference between the pixel 
value of the pixel E and that of the pixel F. When the pixel values of neighboring 
pixels are closer to each other, these pixels may be said to have higher correlation. 
That is, a smaller value of the difference between pixel values indicate stronger 
correlation. 

So, if the difference between the pixel value of the pixel D and that of the pixel 
E, or the difference between the pixel value of the pixel E and that of the pixel F, is 
directly used as a correlation value, the correlation value, which is the smaller 
difference value, exhibits stronger correlation. 

For example, if the correlation between the pixel value of the pixel D and that 
of the pixel E are stronger than that between the pixel value of the pixel E and that of 
the pixel F, the pixel value generator 702 divides the pixel value of the pixel D by 2 
to use the resulting value as pixel data of the area i. 
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Based on the pixel value of the pixel E and on the pixel value of the area i, the 
pixel value generator 702 calculates the pixel values of the pixel data of the area j in 
accordance with the equation (75) or (76), as shown in Fig. 8 ID. 

The pixel value generator 702 calculates the pixel values of the pixel data of the 
area g and those of the pixel data of the area h, for e.g., the pixel D, to calculate the 
pixel value of the pixel data of the area i and the pixel value of the pixel data of the 
area j, and then to calculate the pixel value of the pixel data of the area k and the pixel 
value of the pixel data of vhe area 1, and so on, to calculate the pixel values of the pixel 
data in the picture as described above to generate a horizontal double-density picture 
comprehending the pixel values of the pixel data calculated to furnish the so-generated 
horizontal double-density picture to the frame memory 704. 

Similarly to the pixel value generator 702, the pixel value generating unit 705 
calculates, from the correlation of the pixel values of three vertically arrayed pixels of 
the horizontal double-density picture, supplied from the correlation calculating unit 
706, and from the pixel vfiues of the three pixels, the pixel values of the picture data 
corresponding to two areas obtained on vertically splitting the light reception area of 
the pixel, to thereby generate the double density picture. 

When fed with the picture shown as an example in Fig. 82, the pixel value 
generating unit 702 generates a double-density picture shown as an example in Fig. 83. 

When fed with a picture, shown as an example in Fig.82, the pixel value 
generating unit 705 generates a picture, shown as an example in Fig. 84. When fed 
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with a horizontal double-density picture, shown as an example in Fig.83, the pixel 
value generating unit 705 generates a double-density picture, shown as an example in 
Fig.85. 

Fig.86 is a flowchart for ulustrating the processing for generating the double- 
density picture by the signal processor 12, a structure of which is shown in Fig. 78. At 
step S60I, the signal processor 12 acquires an input picture to store it in the frame 
memory 701. 

At step S602, the relation calculating unit 703 selects one of the pixels in the 
picture as a considered pixel, and Fmds a pixel horizontally neighboring to the 
considered pixel, based on the pixel value stored in the frame memory 701. At step 
S603, the pixel value generator 702 generates a pixel value of pixel data lying on one 
side of the horizontal double-density picture from a pixel value exhibiting stronger 
correlation, that is a highjr correlation value. 

Based on the characteristics of the CCD, the pixel value generator 702 at step 
S604 generates pixel values of other pixel data of the horizontal double-density 
picture. Specifically, the pixel value generator 702 calculates pixel values of the other 
picture data of the horizontal double-density picture, based on the pixel value 
calculated by the processing of step S603 and on the pixel value of the picture data of 
the input picture, in accordance with the equations (75) and (76) explained with 
reference to Fig. 80. The picture data of the horizontal double-density picture for the 
considered pixel, generated by the processing at steps S603 and S604, are stored in the 
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frame memory 704. 

At step S605, the pixel value generator 702 checks whether or not the 
processing of the entire picture has come to a close. If it is determined that the 
processing of the entire picture has come to a close, the program reverts to step S602 
to select the next pixel as the considered pixel to repeat the processing of generating 
the horizontal double-density picture. 

If it is determined at step S605 that the processing of the entire picture has come 
to a close, the correlation calculating unit 706 selects one of the pixels in the picture 
as the considered pixel i:o find the correlation value of the pixel neighboring to the 
considered pixel in the v ertical direction based on the pixel value of the horizontal 
double-density picture stored in the frame memory 704. At step S607, the pixel value 
generating unit 705 generates the pixel value of the one side of the double- density 
picture from the pixel values of the stronger correlation, based on the correlation value 
supplied from the correlation calculating unit 706. 

At step S608, as at step S604, the pixel value generating unit 705 generates the 
other pixel value of the double-density picture, based on the characteristics of the 
CCD. Specifically, the pixel value generator 702 calculates the pixel values of the 
other picture data of the double-density picture, based on the pixel values calculated 
by the processing at step S607 and on the pixel value of the pixel data of the horizontal 
double-density picture, in accordance with the equations (75) and (76) explained with 
reference to Fig. 80. 



PACE 5/38 * RCVD AT 7/18/2009 1:31:02 PM [Eastern Daylight Time] * 6VR:USPTO-EFXRF-6/32 ' DNIS: 2738300 • C6ID:1212S880500 • DURATION <mm-ss):06-12 



07/16/2009 13:32 FAX 12125880500 FROMMER LAWRENCE & HAUG 1)006/038 



155 

At step S609, ths pixel value generating unit 705 decides whether or not the 
processing of the entire picture has been finished. If it is decided that the processing 
of the entire picture has not been finished, the program reverts to step S606 to select 
the next pixel as the considered pixel to repeat the processing of generating the 
double-density picture. 

If it is decided at step S609 that the processing of the entire picture has been 
finished, the pixel value generating unit 705 outputs the so-generated double-density 
picture to complete the processing. 

In this manner, a double-density picture, the number of pixels of which in the 
vertical and in the horizontal directions are doubled, may be produced from the input 
picture by the signal processor 12, the structure of which is shown in Fig. 78. 

The signal process or 12, the structure of which is shown in Fig. 78, is able to 
generate a picture of high spatial resolution by performing signal processing taking 
account of the pixel correlation and the integrating effect of the CCD with respect to 
the space. 

In the foregoing, a picture of the real space having a three-dimensional space 
and the time axis informal ion is mapped on a time space having the two-dimensional 
space and the time axis information using a video camera. The present invention is, 
however, not limited to this: embodiment and may be applied to correction of distortion 
caused by projecting the :5rst information of a higher order first dimension to the 
lower-order second dimension, extraction of the significant information or to synthesis 
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of more spontaneous pictures. 

The sensor 1 1 may also be a sensor exemplified by, for example, a BBD (bucket 
brigard device), a CID (charge injection device) or a CDD (charge priming device), 
without being limited to a CCD. The sensor 11 may also be a sensor in which 
detection devices are an anged in a row instead of in a matrix. 

The recording medium, having recorded thereon a program for executing the 
signal processing of the present invention may not only be constructed by a package 
medium, distributed to users for furnishing the program separately from a computer, 
inclusive ofa magnetic disc 51, such as a floppy disc, having the program pre-recorded 
thereon, an optical disc 52, such as CD-ROM, Compact Disc, read-only memory or 
DVD (digital versatile disc), a magneto-optical disc 53, such as MD (mini-disc) or a . 
semiconductor memory 54. Butmay also be constructed by a ROM 22 furnished to the 
user in a pre-assembled state in a computer, and having the program recorded thereon, 
and a hard disc included in the memory unit 28. 

It should be noted that, in the present specification, the step for stating a 
program recorded on a. recording medium includes not only the processing carried out 
chronologicaUyinthespecifiedsequencebut also the processing that is not necessarily 
processed chronologically but is executed in parallel or batch-wise. 

Thus, based on the *xea information specifying a foreground area made up only 
of foreground object components making up a foreground object in the picture data, 
a background area made up only of background object components making up a 
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background object in th* picture data, and on a mixed area which is a mixture of the 
foreground object components and the background object components in the picture 
data, and on the picture data, the mixed area including a covered background area 
formed at a leading end in the movement direction of the foreground object, and an 
uncovered background ^ea formed at a trailing end of the foreground object, a 
processing unit made up of pixel data lying on at least a straight line extending in a 
direction coincident with the direction of movement of the foreground object from an 
outer end of the covered b ackground area to an outer end of the uncovered background 
area, centered about the f sreground area, are set. Anormal equation is then generated 
by settingpixel values of pixels in the processing unit decided based on the processing 
unit and an unknown dividing value obtained on dividing the foreground object 
components in the mixec. area with a predetermined dividing number. This normal 
equation is solved by die least square method to generate foreground object 
components adjusted for ihe quantity of movement blurring to adjust the movement 
blurring quantity. 

Also, sample data present in detection data lying ahead and at back of the 
considered detection data where there exist considered sample data which is the 
sample data under considei-ation is extracted as foreground sample data corresponding 
to the foreground in the real world, whilst sample data present in detection data lying 
ahead and at back of the considered detection data where there exist considered 
sample data which is the sample data under consideration is extracted as background 
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sample data correspond tog to the background in the real world, and the mixing ratio 
of the considered sample data is detected based on the considered sample data, 
foreground sample data and on the background sample data, thereby enabling the 
detection of the mixing ratio. 

The still/movemeiit decision is given, based on the detection data, and a mixed 
area containing sample data comprised of a mixture of plural objects in the real world 
is detected, thereby enabling detection of the mixing ratio. 

The second signal of a second dimension is acquired by detecting the first signal 
of the real world having a first dimension as mapped on the sensor, with the second 
dimension being lower thim the first dimension, and the signal processing is performed 
on the second signal, thereby enabling the significant information buried due to 
projection to be extracted from the second signal. 

Since the second si gnal is acquired by detecting the first signal of the real world 
having the first dimension by the sensor, with the second signal being of a second 
dimension lower than the first dimension and presenting distortion relative to the first 
signal, and a third signal, alleviated in distortion relative to the second signal, is 
generated by processing based on the second signal, it is possible to alleviate the signal 
distortion. 

In the detection signal, the foreground area, composed only of the foreground 
object components, constituting the foreground object, the background area, composed 
only of the background object components, constituting the background object, and 
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the mixed area compose d of the foreground object components and the background 
object components, are specified, the mixing ratio of the foreground object 
components and background object components at least in the mixed area is detected, 
and the foreground object components and background object components are 
separated from each other based on the specified results and on the mixing ratio, thus 
enabling utilization of the foreground and background objects as data of higher quality. 

In the detection signal, the foreground area, composed only of the foreground 
object components, constituting the foreground object, the background area, composed 
only of the background object components, constituting the background object, and 
the mixed area composed of the foreground object components and the background 
object components, are specified, and the mixing ratio of the foreground and 
background object components in at least the mixed area is determined based on the 
specified results, thus enabling detection of the mixing ratio as the significant 
information. 

The mixing ratio of the foreground and background object components in the 
mixed area comprised of a mixture of the foreground object components, constituting 
the foreground object, and the background object components, constituting the 
background object, is detected, and the foreground and background object components 
are separated from each other based on the mixing ratio, thus enabling utilization of 
the foreground and background objects as data of higher quality. 
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